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ABSTRACT 

This document summarizes the feedback gathered during the user-testing phase in the 
development of an electronic library application: the Aeronautics and Space Access Pages 
(ASAP). It first provides some historical background on the NASA Scientific and Technical 
Information (STI) program and its efforts to enhance the services it offers the aerospace 
community. Following a brief overview of the ASAP project, it reviews the results of an online 
user survey, and from die lessons learned therein, outlines direction for future development of the 
project. 

INTRODUCTION 

The NASA Scientific and Technical Information program (STI) provides a number of 
services and products to the air and space community. One of the most important of those 
products is the NASA STI Database - a bibliographic database of NASA and non-NASA 
aerospace information consisting of more than 3 million records. In 1998, the STI Program 
initiated an overhaul of this database which was to include 

• a more precise standard of cataloging and displaying metadata 

• a web-based graphical user interface with appropriate security controls; and 

• access to available document images. 

A sixty-day period of user testing was held between August and October of 1999. The results of 
and insights gained from this test are the subject of this report. 


BACKGROUND 

The NASA STI Program, through its processing facility at the Center for AeroSpace 
Information (CASI) maintains a repository of NASA and other technical literature, and 
administers a 3 million record bibliographic database representing this and other air and space 
research. This collection dates back to the early part of the twentieth century and is arguably the 
world’s foremost source of air and space research. While the database has existed for over 35 
years, its hardware and software platforms have been migrated and updated numerous times in 
order to keep up with changes in information technology. 

One important technological improvement took place in 1995, when the CASI introduced 
document imaging capabilities to its facility. An electronic document management system 
(EDMS) was developed and entered production phase in 1996. Since that time, all literature 
received at the facility has been stored digitally using the standard known as Tagged Image File 
Format (TIFF). 

The most recent modernization began in late 1997 with the development of a pair of 
prototypical web-based interfaces. One used the contents and database management system of die 
STI program’s production system (a first generation client-server application known as 
RECONplus), replacing its character-based interface with a web front-end. The second prototype 
converted data contained in an older, mainframe-based database implementation (known as 
RECON) to a new server platform, again providing a web-based user interface. 

Members of the NASA STI community tested both these prototypes during die following 
year. While each system had its merits, neither was found to be completely satisfactory. The 
RECONplus implementation lacked a back reference mechanism (a strategy which many 
professional searchers rely upon for narrowing searches) and required a fairly substantial software 
upgrade. The RECON derivative, because its underlying structure had no mechanism for storing 
EDMS metadata, could not be integrated with the TIFF document images. Neither system 
provided an effective means for limiting access on a record-level basis. 



It was thus decided to develop a hybrid system that would incorporate the best features of 

the two prototypes. The development team initiated a number of tasks in order to create this 

hybrid. 

• Data Correction : Existing data records were reviewed with respect to current CASI 
cataloging standards, and nearly 250,000 field values were modified. 

• System Maintenance : The operating system was upgraded and current security and 
maintenance patches were applied. The latest versions of server and database software were 
installed. Additional disk storage was acquired and configured to support the requirements for 
database conversion and storage. 

• Data Migration : The entire contents of the RECONplus database were migrated to ASAP’s 
database management system, while retaining the Machine Readable Cataloging (MARC) 
field definitions used in RECONplus. 

• Access Control : Within the confines of the user test, access control was enforced by means of 
TCP/IP-based address checking. Since testers included NASA civil servants and contractors, 
roughly 37 1 ,000 data records with significant access restrictions were excluded from the test 
database. Within the production release of ASAP, it is expected that a more complex access 
control mechanism will be operational, in order to implement record-level security on a user- 
by-user basis. 

• Access to Document Images : An interface to the CASI EDMS was constmcted to permit 
document images to be displayed upon successful search. 

Work began in April of 1999 with user testing scheduled for late that summer. 


TEST SETUP & PROCEDURES 

The objective of the test was to evaluate the look and feel of the database searching and 
document displaying capabilities of the web-based ASAP database system. To accomplish this, 
the STI Program Office established a point-of-contact (POC) at major NASA facilities. The POC 
recruited ten to fifteen volunteers (engineers, scientists, technical managers and library 
professionals). Testers were granted complete access to the ASAP database throughout the 
duration of the test period (August 23 - October 22, 1999). This included use of the search 
engine and online thesaurus, access to document images, as well as online help on a variety of 
topics. 

Testers were offered the opportunity to comment upon the system by filling out a brief, 
online survey. The survey consisted of three parts: 

I. Respondents were asked to choose the best description of their work (Scientific, 
Engineering, Information, Management, or other). 

II. Respondents were asked to evaluate 17 comments about ASAP, on a scale of 1 (Strongly 
Disagree) to 5 (Strongly Agree). A response of “No opinion” was also allowed. 

III. Respondents were given the opportunity to make additional, open-ended comments by 
means of a text box. 

Ad hoc feedback in the form of electronic mail was received by various areas of the STI 
Program and included among the evaluations. 

With few exceptions, the functionality and design of the system was “frozen” during the 
testing period. Although a number of enhancements could have been added to the system during 
these two months, it was felt that the validity of the user test could best be upheld by insuring that 
all testers reviewed the same version of the system. Given the rather lengthy duration of the testing 
period, combined with the relative ease in development that an internet-based interface allows, we 
feared that otherwise, those who visited the system at the end of the testing period might have seen 
a substantially different implementation than those who viewed it in the early days of testing. 
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Although system development was on hold, data records were added weekly to the ASAP 
database. During the testing period, the CASI processed roughly 22,000 new citations, all of 
which were inserted in the database. This served as a test of some of the system’s internal 
maintenance processes, simulating conditions to be expected within a live, production 
environment. Thus, by the end of the testing period, the database had grown 1%, to a total of 
2,800,000 citations. Approximately 2600 additional images were added to the system during this 
time, increasing die population of available document images by 4%, to a total of 64,698. No 
appreciable increase in system response time was noted, so the inference can be made that the 
system is sized appropriately for at least the marginal growth it will undergo in the foreseeable 
future. 

The development team made a supreme effort to provide feedback to any test participant 
who requested it. These communication efforts resulted in roughly two dozen electronic mail 
and/or telephone conversations between testers and members of the product support team; most of 
these took place within several days after the tester had completed the online survey. This 
interchange often helped the development team by clarifying initial feedback, and, it is hoped, 
benefited the test participants by furnishing instruction on the use of some of the system’s features. 


RESPONSE 

At the conclusion of the two month testing period, the online survey had collected 
feedback from fifty-one different testers. 1 The results of the first section were compiled by review 
of survey logs and can be found in table 1 A. The data is interesting as it indicates that the typical 
desktop environment among the testers is either a Windows NT or Windows 95 workstation using 
Netscape Communicator / Navigator. Roughly 10% of the group used Apple platforms, and the 
same percentage (though not the same individuals) used Microsoft’s browser. 


Workstation 
Operating System 

Netscape 

Navigator 

Microsoft 

Explorer 

Total 

Apple Macintosh 

6 

0 

6 

Microsoft Windows 95 

20 

0 

20 

Microsoft Windows 98 

1 

1 

2 

Microsoft Windows NT 

18 

5 

23 

Total 

45 

6 

51 


Table 1A: Workstation Profile 


Sections I and II of the online survey were compiled by means of a perl-based script. A 
frequency count of the nature of work appears in Table IB. Respondents most often described 
their role as information professional, with quite a few scientific and engineering workers as well. 


1 The complete survey data, in its raw form, can be found online at 
http://www.sti.nasa.gov/ASAP/CASI/user test results.html 
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Thus, we get a thumbnail sketch of the test group. 

Its typical member is a science, engineering, or information 
professional accessing the system via a Netscape browser 
on a 32-bit Windows / Intel platform. Since the testers were 
taken from the population of potential ASAP users, it would 
not be inappropriate to infer that the same configuration will 
hold for all (NASA) users of the system. 

Now that we have a profile of the user community, we turn our attention to their opinions 
about the system that they evaluated. In the first question of Section II, respondents were asked 
to evaluate the statement 'My overall impression of ASAP is positive'. This was 
intended as a general measure of the system as a whole; the average response was 3.5, which 
would seem to indicate that the opinion of ASAP was moderately favorable. Although this result 
is somewhat disappointing, it is worth noting that only 23% of those responding actually 
disagreed with the statement. 

The next three questions had to do with various facets of the usability of the system. 
Responses to each averaged 3.8, from which we make the inference that the users seemed to be 
relatively satisfied with the system’s general design and performance. 

Questions six through eight covered the image display feature of ASAP. Among this 
group was the statement that, at a mean of 4.03, received the highest approval rating within the 
survey: “I like the image display feature and would like to see more document images made 
available”. It comes as little surprise to the project team that the ability to obtain full documents 
in addition to metadata appears to be the most exciting feature of ASAP. Rather disappointing is 
the lack of popularity of, and difficulty of providing support for, TIFF as the standard image file 
fomiat. (Questions 8 and 9). Questions eleven through seventeen sought to gauge the popularity 
of a number of ASAP’s dictionary features. While the average response for each was mildly 
positive, there was little variation. 

A complete listing of the seventeen survey statements of section two, along with statistics 
for each, can be found in Table 2. Mean values were calculated based on a scale of 1 representing 
strong disagreement through 5 representing strong agreement. 


Nature of Work 

Count 

Clerical 

1 

Engineering 

13 

Information Professional 

23 

Scientific 

13 

Teclmical Management 

1 


Table IB: Nature of Work 


# 

Survey Statement 

#Responses 

Agree 

Disagree 

Mean 

1 

My overall impression of ASAP is positive 

42 

62% 

23% 

3.52 

2 

The ASAP system is easy to use and I can perform simple searches 
with it 

42 

81% 

17% 

3.83 

3 

There is a sufficient variety of fields on which to search 

41 

78% 

10% 

3.83 

4 

Response time is adequate and comparable to other similar web sites 

42 

74% 

10% 

3.79 

5 

The results of my search provide me adequate information with which 
to locate report documents in my area of interest 

43 

58% 

21% 

3.35 

6 

Using ASAP helps me to articulate to librarians about my research 
and technical infoimation needs 

37 

59% 

19% 

3.41 

7 

I like the image display feature and would like to see more document 
images made available 

36 

72% 

8% 

4.03 

8 

I had no trouble configuring TIFF viewers and/or viewing the TIFF 
document images 

38 

32% 

53% 

2.58 

9 

I prefer viewing images in PDF to viewing images in TIFF format 

39 

62% 

13% 

3.87 

10 

The Search tips ’and other documentation are useful hints 

36 

64% 

17% 

3.50 

11 

I found the fase NASA Thesaurus in Search’ feature to be useful 

29 

38% 

17% 

3.17 

12 

I found it useful to specify ’records per page’ 

36 

47% 

17% 

3.39 

13 

I found it useful to turn off the ’Search for Plurals’ feature 

29 

34% 

21% 

3.07 

14 

I found it useful to limit searches to available document images 

29 

38% 

17% 

3.24 

15 

I found it useful to browse the NASA Thesaurus 

31 

48% 

19% 

3.39 

16 

I found it useful to browse the dictionary 

30 

47% 

17% 

3.33 

17 

I found the ’sort by relevancy’ feature useful 

34 

41% 

18% 

3.26 


Table 2: Statement Ratings 
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About three-fourths of the testers opted to take advantage of section III of the online survey, 
where they could leave additional, open-ended comments. Quite a few statements were 
complimentary in nature, to wit: 

• “This looks like it will be very useful.” 

• “. . .it is a useful site and allows access to reports which are difficult to locate otherwise.” 

• “Useful - faster than CASI has been - please keep” 

• “I was pleased with the selection of papers that the searches reUimed.” 


But the majority provided what the 
development team likes to think of as constructive 
criticism. In order to provide soundness to the 
analysis of these comments, each submission was 
reviewed independently by two members of the 
project team, who attempted to identify all 
specific criticisms and suggestions, and categorize 
them into general areas for development. The 
resulting efforts were compared and consolidated. 
A high-level summary of the most frequently 
recurring themes is presented in Table 3. It is clear 
from Table 3 that the most frequently occurring 
comments generally concerned searching the 
database. However, in examining them more 
granularly, we see a variety of specific complaints 
(see Table 4). 


Category 

# of Comments 

Search functions 

31 

Page Layout 

24 

Image Viewing 

24 

Metadata Display 

14 

Online Help 

8 

Thesaurus 

7 

Search Results Display 

6 

Response Time 

5 

Database Content 

5 

Site navigation 

3 

Document purchase 

3 

Back referencing 

3 

Access restrictions 

3 

Miscellaneous comments 

21 


Table 3: High level summary of comments 


Search functions 

# of Comments 

Received unexpected/inconect results 

8 

Problems specifying authors using initials 

4 

Lacked support for date and numeric searching 

4 

Lacked browse function 

4 

Miscellaneous negative comments 

4 

Miscellaneous positive comments 

3 

Used Boolean OR instead of AND as default 

2 

Use of Wildcards (“*”) 

2 


Table 4: Specific comments about search functionality 


In most cases, unexpected results seemed to have been caused by the way in which the user 
interface performed Boolean searches combined with its failure to index “stopwords” (in this 
case, common prepositions, conjunctions, and other parts of speech). Including such stopwords 
within a Boolean “AND” search invariably failed to bring back any citations. A similar problem 
occurred when users attempted to specify author names using initials for first and/or middle 
names, because punctuation such as periods (“.”) placed at the end of abbreviations such as 
initials was also removed from the index. Also relevant to this topic is the fact that multiple 
words, when entered within the same input box, were logically “OR'ed”together; this provoked 
comments from several who expected the “AND” operation to be the default. This was a design 
choice made to parallel the behavior of other web-based search engines such as AltaVista, Yahoo, 
and Webcrawler. 
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Several respondents requested that for date and numeric data, there be additional search 
capabilities (such as greater than, less than, between). This was a feature left out of the original 
prototype, quite frankly, due to time constraints. 

A number of people felt this system needed a browse function, wherein a user would be able 
to examine the breadth of values contained within a given database field (something akin to 
“select distinct” feature within SQL). Unfortunately, this is a feature that would be somewhat 
more challenging to implement within ASAP, due to the non-relational nature of its underlying 
database system. 

Another frequently commented upon subject that appears high up in Table 3 was the 
interface’s page layout. Here, there were two major concerns; the first, reported by 24 
respondents, was that the system’s HTML pages, particularly the search screen, tended to be too 
spread out. A number of specific recommendations were made to reduce the amount of white 
space and thus compress the pages into a size more easily navigable within a conventional 
desktop. Secondly, 18% noted some form of inconsistency in the use of the system’s navigation 
icons. 

The one category in Table 3 that generated the most uniform consensus related to the 
system’s document image viewing feature. Many volunteered that Adobe Portable Document 
Format (PDF) was their preferred image file format. Others noted that the major obstacle to their 
use of the TIFF format was the relative scarcity of TIFF viewers and/or difficulty in configuring 
them for use within a Web browser. Whatever the case, nearly half the respondents indicated that 
PDF would be their choice; no one expressed a preference for the TIFF format. 

The category “metadata display” covers issues relating to ASAP’s presentation of 
bibliographic citations. The majority (eight of fourteen) of these comments contained suggestions 
on how to better present or organize this information. There were also several requests for a 
feature whereby users could customize the look of this screen in order to more easily download or 
print out the results. 

Lastly, there were quite a few miscellaneous comments. A number of testers used section 
three of the survey to address questions about the direction and time frame of the project. 
Wherever possible, the test team followed up with electronic mail communication and tried to 
address these issues. It is hoped that the discussion and conclusions that follow this section will 
further shed light on some of these topics. 


LESSONS NOTED / FUTURE DIRECTION 

It has been said that in the aftermath of certain projects, the phrase “lessons learned” is a 
misnomer. Too often, the lessons are noted, but never truly applied. With that in mind, we will be 
conservative with our section heading, but optimistic that members of the project team and other 
readers will indeed make good use of the information herein. 

One surprise was the consistency of system configuration among the testers. Nearly 90% of 
survey respondents were using some fomi of the Netscape Navigator browser; a similar number 
were using a Microsoft Windows operating system. While we make no claim as to whether this 
data is a representative sample of the entire population of NASA workstations, it would seem 
prudent to keep the Windows / Navigator combination in mind when designing and testing Web 
applications for use in NASA. 

Noted as well were the comments regarding page design and layout. In an effort to add to the 
user interface as many features as possible, not enough care was given to human-computer 
interaction. Factors such as the number of mouse movements or keystrokes necessary to quickly 
enter and submit a query should be considered; care should be taken to ensure that actions 
common to all pages are handled identically throughout the system. Efforts are underway to 
improve consistency and compress the layout via use of icons and a reduced-font and re-ordered 
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search screen. The look of the site when it is ultimately released in production will be 
significantly changed. 

Searching was the category which yielded the highest number of comments. Many noted 
unusual or unexpected behavior in the search engine. We see this as an opportunity for enhanced 
user training and support. Past applications introduced at the Center for AeroSpace Information 
have typically required that instruction be focused on the navigation of the system; given ASAP’s 
simple Web interface, we expect that, relatively speaking, a much higher proportion of training 
energy and resources can be devoted to techniques for effective searching. 

Probably the most vocal comments were reserved for problems related to the document image 
viewing feature. The choice of the TIFF image format was made a number of years ago as it was 
a requirement for a number of systems within CASI. It had been hoped that recent advances in 
integrating support for file formats would make TIFF files easily readable on all client 
workstations. The ASAP test has clearly demonstrated that this is not so. The development team 
feels that image viewing is ASAP’s most important and unique feature, and intends to focus on 
this in the near future. Although Adobe PDF is supported by most CASI processing systems, its 
use in the document management modules is limited. We see the introduction of file format 
conversion as becoming routine within CASI document processing systems. 

In addition to digesting the important information contained within the testers’ comments; we 
derive some value from identifying things not said. For instance, there were pleasantly few 
reports of system outages, an issue of concern when the application is being shared nationwide. 
There was some controversy over system response time; while three individuals noted some 
sluggishness in the system, two others took the opportunity to comment that they felt 
performance was acceptable. Moreover, question four of the survey’s section two addressed 
response time directly; the average response was a relatively high 3.79. 

Another topic that received only a single comment was the quality of the bibliographic 
metadata. One survey response noted some flaws within several of the oldest records stored 
within the system, and others inquired as to the meaning or interpretation of specific data fields. 
However, in contrast to the program’s experience with the development of RECONplus, no one 
reported glaring inaccuracies or other problems within the data itself. We take this as vindication 
for exhaustive efforts made in the last several years at CASI to review and revise its MARC- 
based database content, format, and structure. 

The MARC-like structure that serves as the underlying structure for the STI database uses 
nearly 200 distinct data elements. Some of the more esoteric fields - while still contained within 
the database - were omitted from ASAP’s record display form. It is worth noting that the test 
group did not seem to miss this metadata. 

Also missing from ASAP is a full-text search capability. The TIFF specification describes an 
image format, and so does not lend itself to character or word searching. We are unsure how to 
interpret the fact that there were no comments regarding the lack of full-text searching features. It 
could be that this is not perceived as a valuable feature, given the wealth of metadata (including 
full-text abstracts) searchable in the database. It might also be that because the testers had a great 
deal of trouble with the image viewing features in general, the issue of searching within or across 
documents simply did not arise. 


CONCLUSION 

A summary and complete listing of all user survey responses is available online at 
http://www.casi.sti.nasa.gov/ASAP/bin/survev/ADMIN/results.cgi . 

The ASAP user test served two major purposes: it provided assistance to the development 
team in checking out the system’s basic elements, and it offered air opportunity for the STI user 
community to provide input regarding additional features. 
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In its first objective, we feel the test was extremely successful. There were no outages or 
reports of system instability, other than during of one twelve hour period early on the third day of 
the test. (On that day, some users reported that the search function was timing out. This was 
investigated by the development team and resolved by means of a system configuration change). 
Moreover, with the notable exception of the problems with TIFF viewers, each of the components 
of ASAP performed as expected. Lastly, the active presence of members of the test community 
increased the overall load on the system, adding valuable information with regard to the 
development team’s efforts to measure system capacity. 

With respect to the second objective, the development team feels that it gained much 
valuable feedback from potential users of ASAP. Some of the suggestions, though valuable, will 
have to be incorporated into a future version of the system. But the results of others should be 
apparent in the initial production release of the system, which took place in January, 2000. Those 
interested in the end result are invited to visit the project’s web site, at 
http://www.sti.nasa.gov/ASAP . 
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